Single and partitioned variants for datafusion-vortex#1012
Conversation
# Conflicts: # datafusion-vortex-partitioned/benchmark.sh # datafusion-vortex/benchmark.sh
GitHub Actions
Inactive
…ioned (c6a.4xlarge)
|
Results for Logs:
|
GitHub Actions
Inactive
…ioned (c6a.2xlarge, c6a.4xlarge, c6a.metal, c6a.xlarge, c7a.metal-48xl, c8g.4xlarge)
|
Results for Logs:
|
…ioned (c6a.xlarge, c8g.metal-48xl)
|
Results for Logs:
|
|
The run of Logs:
|
|
Thanks @alexey-milovidov! I apologize for not enabling edits by maintainers, I'll make sure to have it enabled next time. |
2 participants
Supersedes #847 by @AdamGS (the fork on that PR has "Allow edits from maintainers" disabled, so the branch could not be updated in place).
This is @AdamGS's
adamg/datafusion-vortexbranch merged up to date withmain, with the merge conflicts resolved.Reintroduces the
datafusion-vortexanddatafusion-vortex-partitionedentries with a much more stable interface that is much closer to the existingdatafusionones, using the up-to-date DataFusion and Vortex releases and a new dedicatedvortex-datafusion-clitool.Conflict resolution
Two files conflicted with
main, bothbenchmark.sh:download-hits-parquet-single/download-hits-parquet-partitioned) and converts to Vortex inload, instead of the old driver handling its own download.mainhad added the single-process concurrent-QPS skip (BENCH_CONCURRENT_DURATION="${BENCH_CONCURRENT_DURATION:-0}", see concurrent_qps measures swap-thrashing, not throughput, for single-process engines (introduced in #860) #946).The resolution keeps both: the shared download script from the PR plus the concurrent-QPS skip from
main. The result matches theduckdb-vortexsibling'sbenchmark.shexactly (the canonical single-process Vortex form), differing only in the download-script name.queries.sqlauto-merged cleanly, preserving the PR'sto_timestamp_seconds("EventTime")wrapping.🤖 Generated with Claude Code